SPEC Changes Summary

Builder

• Added cleanup functions: clear/reset pending messages on startup
• New flag: --skip-clear-pending-on-startup
• Introduced build signature hashing to reuse artifacts
• Builds now pushed to arch-specific streams

CLI

• Renamed --build_arch → --arch (default: amd64)
• Added --command-regex to filter tests by command
• CLI work also sent to arch-specific streams

Runner

• Added --commands-regex and --uri options
• Redis URI parsing (redis://, rediss://)
• Benchmark metric validation (min QPS, latency > 0)
• Reduced log verbosity (info → debug in some cases)

Compare

• New CLI options:
  • --extra-filters
  • --use-test-suites-folder
  • --generate-boxplot, --boxplot-output
  • --command-group-regex, --command-regex
• Added quartile stats (P25, P50, P75) for regressions & improvements
• Boxplot generation (via matplotlib)
• Command/group filtering support

Self-Contained Coordinator

• New flags:
  • --exclusive-hardware
  • --http-port, --http-auth-username, --http-auth-password
  • --skip-clear-pending-on-startup
• Embedded HTTP server:
  • GET /ping, GET /containers
  • POST /reset-queue, POST /flush
• Supports exclusive hardware cleanup
• Flush timestamp support (skip old work)
• Container timeout handling

New Test-Suites (Datasets)

1. memtier_benchmark-1Mkeys-string-setget200c-1KiB-pipeline-10
2. memtier_benchmark-playbook-leaderboard-top-100
3. memtier_benchmark-playbook-rate-limiting-lua-100k-sessions
4. memtier_benchmark-playbook-realtime-analytics-membership
5. memtier_benchmark-playbook-realtime-analytics-membership-pipeline-10
6. memtier_benchmark-session-caching-json-100k-sessions